Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FLINK-35891][cdc-connector][paimon] support dynamic bucket. #3499

Merged
merged 3 commits into from
Aug 12, 2024
Merged

[FLINK-35891][cdc-connector][paimon] support dynamic bucket. #3499

merged 3 commits into from
Aug 12, 2024

Conversation

lvyanquan
Copy link
Contributor

support dynamic bucket.

@lvyanquan lvyanquan marked this pull request as draft July 29, 2024 08:20
@leonardBang
Copy link
Contributor

The issue id is wrong, plz correct it

@lvyanquan lvyanquan changed the title [FLINK-35893][cdc-connector][paimon] support dynamic bucket. [FLINK-35891][cdc-connector][paimon] support dynamic bucket. Jul 30, 2024
@lvyanquan lvyanquan marked this pull request as ready for review July 31, 2024 01:44
@lvyanquan lvyanquan force-pushed the Paimon_dynamic_bucket branch 2 times, most recently from 9ad64e1 to 396245a Compare August 1, 2024 15:33
@lvyanquan lvyanquan force-pushed the Paimon_dynamic_bucket branch 2 times, most recently from 9c1fcf9 to 54cabcd Compare August 9, 2024 15:23
@github-actions github-actions bot removed the runtime label Aug 9, 2024
@leonardBang leonardBang self-requested a review August 9, 2024 15:24
yuxiqian
yuxiqian reviewed Aug 12, 2024
View reviewed changes
Copy link
Contributor

@yuxiqian yuxiqian left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for @lvyanquan's great work! Just left some general comments.

flink-cdc-common/src/main/java/org/apache/flink/cdc/common/sink/DataSink.java Show resolved Hide resolved
...tor-paimon/src/main/java/org/apache/flink/cdc/connectors/paimon/sink/PaimonHashFunction.java Show resolved Hide resolved
...rc/main/java/org/apache/flink/cdc/connectors/paimon/sink/v2/bucket/BucketAssignOperator.java Outdated Show resolved Hide resolved
...ain/java/org/apache/flink/cdc/connectors/paimon/sink/v2/bucket/BucketWrapperChangeEvent.java Outdated Show resolved Hide resolved
...ava/org/apache/flink/cdc/connectors/paimon/sink/v2/bucket/BucketWrapperEventKeySelector.java Outdated Show resolved Hide resolved
...aimon/src/main/java/org/apache/flink/cdc/connectors/paimon/sink/v2/bucket/BucketWrapper.java Show resolved Hide resolved
...n/java/org/apache/flink/cdc/connectors/paimon/sink/v2/bucket/BucketWrapperEventTypeInfo.java Outdated Show resolved Hide resolved
...main/java/org/apache/flink/cdc/connectors/paimon/sink/v2/bucket/BucketWrapperFlushEvent.java Outdated Show resolved Hide resolved
...paimon/src/test/java/org/apache/flink/cdc/connectors/paimon/sink/PaimonHashFunctionTest.java Outdated Show resolved Hide resolved
...paimon/src/test/java/org/apache/flink/cdc/connectors/paimon/sink/PaimonHashFunctionTest.java Show resolved Hide resolved
@lvyanquan
Copy link
Contributor Author

Thanks @yuxiqian, addressed your comments.

@lvyanquan
Copy link
Contributor Author

The failed case in mysql is unrelated to the change of this pr.
Error message:

Caused by: java.lang.IllegalStateException: Invalid assigner status {} [NEWLY_ADDED_ASSIGNING_FINISHED]
	at org.apache.flink.util.Preconditions.checkState(Preconditions.java:215) ~[flink-core-1.18.1.jar:1.18.1]
	at org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlSnapshotSplitAssigner.onBinlogSplitUpdated(MySqlSnapshotSplitAssigner.java:481) ~[classes/:?]
	at org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlHybridSplitAssigner.onBinlogSplitUpdated(MySqlHybridSplitAssigner.java:189) ~[classes/:?]
	at org.apache.flink.cdc.connectors.mysql.source.enumerator.MySqlSourceEnumerator.handleSourceEvent(MySqlSourceEnumerator.java:167) ~[classes/:?]
	at org.apache.flink.runtime.source.coordinator.SourceCoordinator.handleSourceEvent(SourceCoordinator.java:590) ~[flink-runtime-1.18.1.jar:1.18.1]
	at org.apache.flink.runtime.source.coordinator.SourceCoordinator.lambda$handleEventFromOperator$3(SourceCoordinator.java:297) ~[flink-runtime-1.18.1.jar:1.18.1]
	at org.apache.flink.runtime.source.coordinator.SourceCoordinator.lambda$runInEventLoop$10(SourceCoordinator.java:469) ~[flink-runtime-1.18.1.jar:1.18.1]
	... 8 more

leonardBang
leonardBang approved these changes Aug 12, 2024
View reviewed changes
Copy link
Contributor

@leonardBang leonardBang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @lvyanquan for the contribution and @yuxiqian for the review, LGTM

@leonardBang
Copy link
Contributor

Merging... The failed case is not related current PR，I‘ll file a issue ticket to track it

@leonardBang leonardBang merged commit 64f996b into apache:master Aug 12, 2024
20 of 21 checks passed
qiaozongmi pushed a commit to qiaozongmi/flink-cdc that referenced this pull request Sep 23, 2024
@lvyanquan @qiaozongmi
[FLINK-35891][pipeline-connector][paimon] Support dynamic bucket in P…
fac72d9 
…aimon sink

This closes  apache#3499.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@leonardBang leonardBang leonardBang approved these changes

@yuxiqian yuxiqian yuxiqian approved these changes

Assignees
No one assigned
Labels
approved common composer paimon-pipeline-connector reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@lvyanquan @leonardBang @yuxiqian